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INTRODUCTION 



Abuse of alcohol and other drugs causes a variety of problems in a community. Some of these 
problems are old and well established: others are newer, although growing at an alarming rate in 
many places. In recent years new kinds of partnerships among the variety of political, social, and 
economic sectors of the community are necessary in order to better understand these problems 
and take broad-based action to effectively intervene. 

Because the effects of alcohol and other drug abuse are felt in so many sectors of the 
community, it is not easy to get comprehensive information reflecting overall community impact. 
School teachers and administrators, for example, may be aware of problems with alcohol and 
drugs which interfere with education in the community. Medical providers may know about health 
problems caused by drugs and alcohol. Police and law enforcement personnel confront criminal 
activity related to alcohol and other drugs on a daily basis. But these isolated pieces of information 
are usually not fed to some central entity responsible for planning and carrying out a community 
response to drug and alcohol problems. 

Those working to establish community partnerships for substance abuse prevention will find it 
important to look for a way to consolidate information frcm several important community systems 
as a step in the planning and development of effective and comprehensive substance abuse 
prevention activities. 

This is a manual for use by a group of citizens working to build a broadly-based community 
alcohol and drng abuse prevention program. Such groups get started in various ways-often 
called together by community leaders representing the schools, local govemment. or business and 
industry. The steps necessary to get su-.h a group started are covered in a number of books and 
manuals which are already available {see Appendix A for some examples). The assumption here is 
that such a group has been formed and has started working. The group will grow and change over 
iime. but the critical early linkages and commitments have already happened. 

Most of the written material about community coal'itions for alcohol and drug abuse prevention 
advises that an early step in the process should be a "needs assessment." That is. these groups 
ought to have data at their disposal which accurately and objectively depict the status of the 
problem in their community. As noted above, however, such a picture is multi-faceted, and the 
process of obtaining it can be a complex endeavor. Traditionally, "needs assessment" involves a 
combination of information gathering tasks, including suiveys. inteiviews with "key informants", 
and collectic- and analysis of indicator data from diverse community sources. Written guidelines 



and directions for gathering and evaiuating existing indicator data tend to be quite generai. 
however, and typically leave readers to their own devices when stumbling blocks are encountered. 
Many of these obstacles are predictable, however, and this handbook is intended to be helpful in 
getting past at least some of them. It includes both a systematic framework which a coalition can 
apply to its community, and a detailed case study of one community group's success with an 
indicator data approach to assessment of drug and alcohol abuse community impact. With this 
framework, the end product in other communities could be similar in format to the one described 
in the case study, or it could be quite different. 

The remainder of this handbook consists of three sections. Part One asserts the need to 
assemble such a profile. The purpose for the data gathering activity must be cleariy articulated. Ii 
Part Two the steps in assembling a package of community indicators are presented, each 
accompanied by an illustration from the Regional Drug Initiative in Portland. Oregon. Finally. Part 
Three suggests guidelines for using the information. 
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Part One 



The Need for a Community Profile 

It is generally understood that abuse of alcohol and other drugs hurts a community and its 
members In a wide variety of ways. Effects are felt In families, sciiools, the workplace, and so on. 
touching nearly every organization and Informal group In existence. In addition, the nature of the 
damage varies widely. Some of the typical adverse Impacts Include: 

- Health damage suffered by users. 

Birth defects, brain damage, etc. in unborn babies. 

- Child abuse and neglect. 

- Fatalities, injuries, and property damage caused by alcohol-impaired and other 
drug-impaired persons. 

Prostitution and theft. 

Lost productivity and on-the-job accidents. 

- Severe emotional distress (frequently leading to life-long consequences) suffered 
by spouses, children, parents, siblings, users, and others. 

With so many social and organizational systems affected. In such a wide variety of ways, it is 
unlikely that any one organization or IndivkJual Is aware of the complete picture. Resources 
dedicated to responding to the problem are typically distributed among many organizations and 
functional sectors in the community, such as the health care system, the justice system, the 
educational system, and so on. Most of these entitles handle alcohol and other drug problems as 
simply one aspect of their overall role. Not surprisingly, a picture of the overall scope of the 
problem and its total Impact on the community does not simply emerge without effort on 
someone's part to pull all the parts and pieces together. 
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A comprehensive view of the impact of alcohol and other drug abuse on a community is useful 
for two purposes: 



1. Tn r^l«? ? awareness . Individuals in the community are likely to be aware of parts of the 
problem, but its overall scope and size is likely to be surprising. Awareness can be a step 
toward action, once individuals come to view the total impact of substance abuse as an 
intolerable (and preventable) situation. 

2. Tn focus the action . Objectivo data and information can guide the planning process as 
decisions are made about aspects of the problem to attack, and types of interventions to 
employ. (Although many existing community resources may be confronting substance 
abuse problems and issues, they typically must devote most of their efforts to coping with 
the existing problem, rather than systematic prevention aimed at keeping future problems 
from occurring.) 



Case Study: Tho RDI Experience 
Exhibit 1 



In Portland, Oregon the Regional Drug Initiative (RDI) Task Force, was formed 
by professional, business, government, and community leaders. The coalition 
has developed a way of Incorporating a typical array of available local data in a 
simple and manageable format which serves both as a tod to increase 
community awareness, and as an aid in planning the coalition's overall 
direction. When the coalition was formed in 19S7, information about the impact 
of alcohol and drug abuse in the community was available from scattered 
sources and reported on an erratic basis. Utile hard information was available 
in £ me key areas, such as dmg impacts in the workplace. Statistics and other 
information on topics such as drug-related crime, dmg related deaths, etc. 
came from a variety of sources. They were not reviewed or processed by a 
single responsible entity which could assemble a composite view of 
community impact. There was an emerging av/areness that substance abuse 
was hurting the community in many vi«ys. and ihat the effects were being felt 
in virtually all economic, population, and organizational sectors, but the 
information needed to organize and conduct community-wkJe prevention was 
lacking or widely scattered. One of the group's first steps was to establish a 
working group with the task of finding a useful overall stmcture for gathering, 
organizing, and sharing existing information. That group's experience and the 
"Drug Impact index" it developed over an approximately 18-month span may 
prove helpful to similar groups in other communities. 
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The method of information gathering described in this handbook reiies soleiy on the use or data 
which already exists within the community. This is in direct contrast to methods which focusj on 
cdiecting new information-surveys or questionnaires, personal or telephone interviews, etc. 
Advantages of tapping existing data include increased potential for obtaining information about 
many aspects of community impact, lower cost and time expenditure, and the ability to Jbserve 
trends in the data right away, since it may be possible to "look back" in time. A new data collection 
instrument or procedure must be administered several times before trends can be detected. With 
the use of existing data, assuming it has been collected consistently for some time, the trend up to 
the current moment is already known. For many potential indicators of the impact of alcohol and 
other drug use in the community, information provided about the direction of the trend may be the 
most illuminating aspect. As we will see, many factors can obscure an indicator's ability to 
estimate the absolute size of an impact at a given moment, but a trend in the indicator over time 
still tells a story. 



Prrt Two 



Building a Package of Community Indicators 

Assembling a list of community Indicators can l>e a long and tedious process. Getting an 
organized start and following a consistent series of steps will help. The following is a series of 
steps local citizens can take to assemble a package of drug abuse impact indicators for a specific 
community. 



Step 1: "Brainstorm" a list of possible indicators 



Convene a working group. Its members should include persons with some knowledge of 
available Information in r. variety of domains (law enforcement, health care, employer-employee 
relations, education, etc.), as well as persons whose orientation is more general--who are more 
likely to see and think about information the way the target audience will see and think about it. The 
working group may be as small as seven or eight, or as large as 12 to 15 persons. Subgroups of 
three to five persons will be needed for specific assignments. 

The working group can start with a "brainstorming" session to g3nerate a list of possible 
indicators of community impacts caused by alcohol an J other drug abuse. A key feature of the 
brainstorming process is that all ideas have squal weight. Ideas are recorded on a list as soon as 
they are suggested. They are not screened according to .Hght vs. wrong, important vs. trivial, 
feasible vs. unfeasible, or any other measure of merit. The intent should be to identify as many 
existing information resources as possible, while at the same time generating ideas about 
information which would be valuable to have, whether or not it is known to be available. 
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Case Study: The RDI Experience 
ExhibK 2 



The Regional Drug Initiative community impact working group originally 
conveneo to consider a task force member's suggestion that the effects of 
substance abuse might be more easily understood if ail the data could 
somehow be boiled dowr^ to a single index "like the consumer price index." 
Such an index would indicate the scale of the problem at a given time and 
show the trerxi of changes over time. The working group, drawn from several 
organizations which collectively had knowledge of a variety of information 
sources, began the task by making a list of types of data known to exist, and 
questions to be answeied if pertinent information sources could be located. 
Although it quickly became apparent that the available yardsticks dkl not have 
enough in common to ccrr,b:r.d them in a single statistic, the group decided 
that a short list of different kinds of indicators, from a variety of pertinent 
sources, would do almost as well for its purposes as a single index. 



For best results. :he group should be fairly comfortable, and up" before attempting 

this task. If members of the group feel shy about speaking up. some of the best ideas may not get 
on the list. A practice, "warm up." exercise in brainstorming ("Favorite fruits and why." for example) 
can be used to get the group going before it attempts the task of building a list of potential 
indicators of substance abuse community impact. 

In beginning the indicator listing task, it might be helpful to start with a description of the single 
composite index idea. Then ask the group to suggest possible elements of such a composite 
index. Every idea, from the obvious to the unusual to the seemingly impossible, should be 
welcomed during this stage of the process. Prioritizing and critical examination will come later. 



Case Study: The RDI Experience 
ExhibHS 



Some the ideas consWered for the RDI Index Included; 

Annual number of drug overdose deaths. 

Annual number of drug-affected births. 

Percent arrestees testing positive for Illegal substance(s). 

Hospital and emergency room admissions for drug and alcohol related 
causes. 

Age of onset of drug use among school children. 

Number of adult felony drug cases issued by the District Attorney's 
office. 

Number of drug and alcohol school suspensions. 
Number of clients treated In publicly funded drug treatment. 
Number of employers with alcohol and drug policies. 
Amount of drugs stolen from hospitals, pharmacies, etc. 
Wumber of identified "drug houses". 

Number of schools offering curriculum focusing on drug and alcohol 
prevention. 

Number of employers using pre-employment drug screening. 
Juvenile arrests related to drugs and alcohol. 
Amount of drug-related crime. 
Amount of gang-related drug activity. 
Citizen Involvement In anti-drug activities. 

Number of children removed from families for alcohol/drug-related 
causes. 
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step 2: Rounding out the field 



Once the "brainstorming" list is assembled, its elements can be examined a iittie more criticaiiy. 
The nexl steps wiii help the group discover gaps in the list of potential indicators, and may lead to 
the listing of additional possibilities. 

The brainstorming session should have generated a variety of information ideas. Some kind of 
classification structure can now be introduced. For example, the suggested indicators might be 
sorted into categories, such shown in the table below. Such a table can be a worksheet for listing 
proposed indicators within categories. 

Exhibit 4 



Category Examples of Potential Indicator(s) 



Health 


Overdose 
deaths 


D'^ug-affected 
births 


AOD-related hospital 
cases 


Education 


Drug-related 
suspcnsiuns 


AOD use survey 
findinos 

III i^^ii i%j^ 


Student hours in 
prevention curriculum 


Treatment 


Enrollments in 
treatment programs 


Average waiting 
list size 


Usage trends by drug 
category 


Justice 


Drug crime 
arrests 


Arrestee drug 
results 


Crime report trends 


Youth 


Juvenile drug 
arrests 


Underage drinking 
reports 


Availability of AOD- 
free recreation and 
social activities 


Families 


AOD-related 
foster care 
placements 


Parent training 
events provided 


AOD-related family 
problems tied to 
unemployment 


Safety 


AOD-related 
traffic 
injuries & 
deaths 


AOD-related 
drownings & other 
accidents 


Discarded syringes, 
etc. in parks & 
playgrounds 


Business 


AOD-relmed 
sick leave 


Productivity & 
product quality 
deficiencies 


Employee Assistance 
Program referrals 


Other 


Community 
improvements 

deferred due 
to AOD impacts 


Tax revenue losses 
due to AOD impacts 


Adverse mecla 
exposure affeciing 
community reputation 



This sorting task can help the group flesh out the list of indicator candidates, and make other 
improvements which the exercise may tend to suggest. 



Step 3. Find Loanable data 



Irwormation relating to the items on the list of potential indicators may be available from a 
variety of sources. Examples of some of these are shown below. 

0 Many civic groups prepare reports on topics of community concern which may contain or 
provide leads to useful information. 

0 Government agencies and non-profit organizations which provide seivices such as health 
care, criminal justice, treatment and rehabilitation, etc.. often maintain statistical profiles of 
the populations they serve and the services they render. 

0 Information about impacts in the private business sector may be available from major 
employers and industry associations. 

0 Academic research conducted in local or nearby colleges, universities or research 
institutes may also be a source of community data. 

0 Local media orten collect and report information related to the impact of substance abuse 
in the community. The local libri..y. or the media organizations themselves, may be a 
source of collected, published material. 

0 Health-related informatioii might be accessible through county or state health 

departments, the state or county medical examiner, hospitals and health maintenance 
organizations, and professional practitioner organizations. 

0 Information about impacts in the education system may be available from local districts, 
stata education agencies, or a regional resource such as the Western Center for Drug-Free 
School and Communities. 

0 Schools and school districts, public and private, may have information relating to 

substance abuse Impacts and positive steps taken (e.g.. children exposed to prevention 
curricula, parents participating in awareness seminars, as well as number and type of 
disciplinary actions taken, security/surveillance measures implemented, etc.). 

O 10 
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0 Criminal justice system data may be available from local law enforcement jurisdictions in 
the form of Uniform Crime Report and arrest data; conviction and incarceration 
infoitnation may be available from the county prosecutor, and other information such as 
the number of convicted drug offenders currently at large under supervision may be 
obtained from local, state, and federal probation and parole authorities. 

Many of the organizations you will need to contact will be listed in the telephone directory 
government listings. Look through the listings for city, county, regional, state, and federal 
agencies, as well as local school districts. Private and nonprofit organizations may be in the 
Yellow Pages under "organizations" or subject headings such as "alcoholism treatment", "drug 
abuse information and treatment." "hum^n service organizations." "social service organizations," 
"youth organizations" and so on. 

The organizational structure of public sector service providers varies from community to 
community, so you will need to do some investigating to find the best sources. Allow plenty of 
time for muiciple phone calls as you track down leads. Be prepared to explain what you want to 
several different people within a given organization before you finally get connected with someone 
who might be able to help you. Workers within service delivery systems will usually do their best to 
help you. but they may be unaware of certain details relating to data and unsure as to whom such 
requests should go. Generally, you will find one key person in an office whose job it is to put the 
data together each month, quarter, or year. When you find this person, celebrate, and nurture the 
contact! Your needs for information from this office will usually be met by this individual. 

You will develop your detective skills as you work your way through these systems. Don't be 
discouraged by Wind alleys and false trails; every detective must deal with these from time to time. 
Always treat the people you speak with in this effort with courtesy and respect. They are not your 
adversaries. They are not deliberately giving you the "runaround." (Since the compilation of the 
information you need is probably g/ 'ng to be extra wori< for s nr^^ne. there is always the 
possibility that the task will be put asWe and neglected, if not t( .gotten. Reminder calls may be 
required, or you may even resort to consulting a higher supervisor. This should be handled in a 
professional nrtanner. without intimidation or rancor.) 

Keep track of your detective woric on each indicator. You might want to use a worksheet such 
as that shown in Exhibit 4.1 (attached). 
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Exhibit 4.1 



Data Collection Worksheet 
(Make copies of this page to use in tracking steps taken for 

each indicator) 

Indicator Name 

Indicator Category 



Date ^Person Contacted 

Organization^ _Phone 

Discussion Summary — 



Date_ Person Contacted_ 

Organization ^ Phone 

Discussion Summary 



Date ;_Person Contacted 

Organization Phone 

Discussion Summary 



ERiC 
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step 4. Screen indicators according to availability 



As you begin and progress through your search, you will find a considerable range in the 
availability of information. You can probably group the results in four general c&tegories of 
availability. 

The first category contains infomiation which Is available in the form you expected or hoped to 
firvj it in. For example, you wanted the number of parents attending a school-sponsored 
substance abuse prevention workshop series, and found that the school district was able to 
provide the information. When this happens, celebrate! But be fcrewamed--you will rarely find 
information in this category of utility. 

The second category contains information which is in some way different from what you 
expected or hoped for. but which may still apply to the concem reflected by the original question. 
For example, you wanted a breakdown of drug-related crimes, but the available data can't answer 
this question directly. Some combination of crime report statistics, arrest report statistics, offense 
history analysis of substance-abusing criminal offenders, and educated guesswork is needed to 
arrive at an estimate of drug related crime. Much of the information you find will be in this 
category. 

The third category contains information you weren't looking for. but which nevertheless seems 
pertinent to the issue of community impact. You might, for example, find out about levels of 
prescription drug abuse, even though your initial questions dealt entirely with alcohol and/or illegal 
substances. This situation will occur frequently and may yield unanticipated benefits in your quest 
to profile yoi r community. A word of caution, however: beware of getting sidetracked with issues 
which are less directly related to the central purpose of the coalition. An illustration of such a 
journey is given in Appendix B. While the problem ot substanca abuse is often entangled with 
many other social problems in a community, and a systematic synthesis of all the existing data 
relating to all these problems might seem necessary, such an effort can cause costly delays in 
meeting the goals of the present project. 

A certain amount of additional information about the community could be helpful at this stage, 
however, in order to place some of the indicator candidates in context. Some basic demographic 
information, like age. sex. ethnicity, income distribution, education, and family structure data for 
the community will help meet this need. Other indicators of community strengths and problems 
(attachment and cohesion indicators such as voter turnout, amount of owner-occupied housing. 
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availability and variety of organized activities, community accomplishments and projects, etc.) 
could also be helpful. 

Finally, you will likely find that information relating to some of your indicator possibilities simply 
does not exist in your community. The business sector, for example, may not know much loss due 
to productivity 'mpairment. theft, and waste Is attributable to substance abuse. As you round out 
your list of Indicators, your group should take note of such gaps in information availability, and 
consider recommending that steps be taken to begin collecting pertinent data. 



Step 5. Look for regional and national comparisons 

Regional and national comparison data related to the potential indicators on your list can help 
show similarities and differences between your community and region and the nation as a whole. 
While the presentation of data and information on your community will spark some interest, the 
inevitable question will follow-"How do we compare to other places?" 

The detective work involved in obtaining comparative data is similar to that required for local 
data. The contacts you make in local organizations will often be able to help you find the best 
regional, state, or national information for comparison with the local information they supplied. 
They may already have the latest available comparative data, since they are likely to be on the 
distribution list for such reports. 
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Case Study: The RDI Experience 
Exhibit 5 



One of the more fmstrating problems encountered by the RDI in the 
construction of its Drug Impact Index was the lacl< of comparable state or 
national data for many of the indicators. Despite the huge variety and amount 
of statistical information collected and maintained by various entities in this 
country, it often turns out that data relating to a particiiar indicator is collected 
and organized in different ways from place to place, preventing direct 
comparison. The RDI group found this to be true in the case of such indicators 
as drug-related deaths (a category in which the state and local data was 
restricted to illegal drug overdoses, while national compilations reflected 
reports from other states which included a wider range of substances). Drug- 
affected births are also reported inconsistantly from place to place, and 
sometimes even from hospital to hospital within a single region. After 
struggling first with tho problem of finding comparison data, and then the 
related problem of devising a way to "leaningfully display the somewhat 
comparable data which was found, the RDI group decided not to use any 
national comparison data at ail in its report to the coalition membership and 
the community. Statewide data was used for srveral indicators, however, 
including overdose deaths, drug-affected births, student drug use. and drug 
offense arrests. 



Step 6. Reconsider the list 



When the first round of data gathering has been completed, the group Is in a position to 
reconsider its proposed list of indicators. The group should meet to prepare a revised list of 
indicator candidates. The revised list may contain items changed from their organizing form to the 
closest approximation which can be answered with available data, new items discovered during 
the data availability research, as well as items for which no change was needed. 



Step 7. Identifying the best indicators 



With the revised list in hand, the group is ready to begin selecting the strongest indicators for 
the final list. Selection criteria should include availability, technical quality, and information value. 
Technical quality refers to validity and reliability, discussed further below. The information value of 
an indicator can be understood as the additional contribution to overall knowledge of community 
impact which that indicator provides. Because it is important to represent a range of different 
kinds of community impact, an indicator which provides information about an area of impact in 
which fev^ indicators are available might be judged to have greater information value than an 
additional indicator for an area for which many indicators already exist. 
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Case Study: The ROI Experience 
Exhibit 6 



There came a point in the RDI Drug Impact Index project when the group 
found it needed expert advice in order to move ahead. Worl<ing on the 
principle that someone, somewhere, had probably already carried out a similar 
project, members of the group went looking for expertise. Perhaps 
surprisingly, they didn't find any other group in any other community engaged 
in quite the same kind of effort. They turned then to a locally available source 
of expertise on uses of social indices and indicators. At this point in the 
development of a Community Profile, it is imperative to consult with someone 
with expertise in statistics, measurement, research, and/or social index 
construction-particulariy at the evaluation stage, when potential indicators are 
screened in or out according to the standards of technical quality and 
information value. 



Technical Quality (Validity & Reliability) 

An indicator is valid if it actually measures what it is meant to measure. An indicator is reliable il 
it is consistent in repeated use overtime, so that changes in the measurements obtained at 
different times accurately reflect actual changes in the thing being measured. For example, if we 
wish to compare the performance of runners in the 100 meter dash, we can do so by determining 
that elapsed time is a valid indicator of the performance characteristic we want to measure, and by 
making use of a reliable timing instrument, such as a stopwatch, to time the runners. To be 
considered reliable, the stopwatch should run at the same constant speed each time it is used, 
independent of outside influences such as temperature, barometric pressure, tension on the 
mainspring, random variation, or the gravitational influence of the moon. As this simple example 
demonstrates, an indicator is valid if it has a demonstrable relationsnip to the quality it purports to 
measure. It is reliable if it provides consistent accuracy in its measurement. 



Validity: Arrest and crime reporting examples 

Validity is a key concern when using pre-existing data obtained from a variety of sources. The 
audience for the community profile you are constructing may not be particulariy concerned with 
this, but if the information is to be used to direct policy and action, you need to be. For example, a 
typical validity issue may arise around the question of indicators of criminal activity. Crime is 
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usually a significant factor in the impact of substance abuse on a community. Yet crime by its 
nature is behavior which is often not rtirectly observed. 

Typical community indicators of criminal activity incJude crime report statistics and arrest rate 
statistics. The validity of each of these as a reflection of the amount of criminal behavior actually 
occurring can be dramatical'/ affected by other factors unrelated to the actual number of crimes 
occurring-such as changing community norms relating to crime reporting, or changes In law 
enforcement response pattems. 

Crime report statistics are compiled from reports provided by citizens. Many factors can 
influence citizen reporting, including the level of public confidence that making the report is worth 
the trouble, the perceived seriousness of the crime, and sometimes the victim's degree of concern 
about possible exposure of his or her part in other illegal activity. If. because of workload or other 
factors, police priorities change such that certain offense categories receive less attention, the 
puWte is less likely to file reports, while the actual number of occurrences may increase due to the 
reduced risk of arrest perceived by offenders. 

Arrest rate statistics are a more limited source of information about some kinds of criminal 
acth'ity than are crime report statistics, because they include only crimes for which an arrest was 
made. Also, changes in law enforcement agency policy and practices can have a more direct and 
immediate effect on arrests than they have on crime reports from the public. On the other hand, 
arrest data is likely to be the only infomiation available for certain categories of crime, including 
drug dealing and prostitution, which are not often reported to the police by a victim or witness. 



Case Study: The RDI Experience 
Exhibtt 7 



Despue their limitations, crime and arrest reports are wkJely used and are likely 
to be important elements of any Communiry Profile. They are the likely to be 
the best available indicator of criminal activity-. The best defense against the 
danger of misreading such data and reaching inappropriate conclusions is a 
thorough knowledge of possible alternative explanations for ooseived results. 
For example, the RDI impact Index task group concluded that a drop in drug 
arrests in one year was the result of a police department reorganization which 
temporarily limited the department's capability to investigate drug cases rather 
than a reflecttori of reduced drug crimes. The Drugs and Vice unit had been 
closed down, and its officers reassigned to other duties by a newly-appointed 
Chief of Police, in the belief that drug cases could be handled by non-specialist 
detectives and offteers. The inforrnant networi< anc' other Inteiligence-gathering 
resources which bad been established and maintained by the Drugs and Vice 
unit did not functton well under the new arrangement, and the number of cases 
dropped significantly. The number of cases returned to prior levels when the 
Drugs and Vice unit was re-established. 
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Reliability 



The reliability of a proposed indicator can be assessed by learning about the history of the data 
collection effort, the methods used in collecting the infonnatlon, and the likelihood that comparable 
information will be available in the fut.^Te. Has the information been collected and reported on a 
regular basis over a period of time? Is a consistent method of collecting, summarizing and 
reporting the information in place and consistently used? Is likely that the effort will continue? 



Cm Study: Tht RDI Experience 
Exhibits 



The RDI group used the reliability criterion to eliminaie some proposed 
indicators, such as estimates of the number of "drug houses" in operation, 
when it was found that a reliable method of obtaining such a count did not 
exist. "Official" estimates provided by various agencies varied widely, due to 
fundamental disagreements over definitions, and the transitory nature of the 
phenomenon itself- a dwelling unit passing quickly into and out of use for 
volume f^Tug dealing. The RDI Indicator list does, however, still contain several 
items with little or no history of availability. These were included either 
because the effort to collect and report the information had already begun, and 
was judged likely to continue, or because the RDI group believed the 
information should be collected and deckJed to advocate for its 
implementation. Advocacy for action related to the problem of drug and 
alcohol abuse in the community, of course, is completely consistent with the 
larger purpose of a community coalition on substance abuse prevention. In 
this case, it was the exarcise of researching and developing its' "Dmg Impact 
Index" which led to the discovery of an opportunity to advocate for an action 
which might enable the community to better understand and deal with its 
substance abuse problems. 



Step 8. Choose the best approximations 



Chances are, there will be some concerns about the valWity and/or reliability of each of your 
indicator candWates. The group's task is simply to deckJe which of the possibilities are the least 
seriously affected by these problems. Also, which Indicators are least likely to be seriously 
misinterpreted out of context? Can the necessary context be provWed in the framework of a 
simple, short report dealing with a range of problem and response indicators? 
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include a range of impact types 



Another important element in the selection process is representation of a range of different 
types of impacts. Domof' Jffects of sul)stance abuse problems in many facets of 
community life is a maj ihe project. The group will need to weigh its judgement of the 

relative strength of a ^ , ,dd indicator in terms of validity and reliability against the importance of 
providing indicator's across a spectmm of types of impact. A given indicator may be viewed as 
relatively ^eak compared to others, yet be the best available in an important area of concem. The 
group can decide that inclu'jion of the indicator is warranted, based on Its judgement of the 
importance of calling attention to the area of concem that It represents. At the same time, the 
importance of that indicator ought to motivate increased commitment to its reliable and valid 
assessment 

The variety of indicators, from different systems and sectors of the community, is one of the 
main strengths of the community impact index. Limitations of some of the indicators are to some 
extent cancelled out by the strengths of others. Taken together, the overall profile index will tend 
to reveal a more complete view of the problam* "^eeks to measure than can be seen from the 
vantage point of any of the indivWuai indicators. With the continuing involvement of appropriate 
representatives of organizations which supply indicator data, greater precision in measurement 
can be expected to evolve over time, as the limitations of the separate indicators are identified, and 
ways are found to improve their accuracy. 



Educational benefits 

At this point in the process, the group should have acquired a good understanding of some of 
the difficulties encountered in attempting to measure substance abuse community impacts. 
Appreciation of the tolerances and limitations inherent in the data, and some of the possible 
alternative explanations for observed results, will prove useful as the group turns its attention to the 
task of disseminating its results and facMftating use of the information in policy and action decisions 
throughout the community. Group members will be in a position to assist information users gain a 
sufficient understanding of the data to minimize misinterpretation due to incorrect and/or 
unexamined assumptions. This is not to say that there should be a single, "correct" interpretation 
of the community impact infonnation. Valid disagreements about the meaning of data, based on 
differing value systems and priorities, are alvi^ys possible and usually healthy. The major purpose 
of the exercise will be accomplished if it leads to increased public interest and support for efforts to 
prevent further community problems resulting from the abuse of alcohol and other dmgs. 
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Part Three 
Using the impact Index 

In order to serve its purpose, the community Index must get attention. An effective format and 
carefully limited quantity of content material wll Increase the chances that the written document 
forni of the index will be read and understood. 



Format Considerations 

The format of the report should be simple anu u'-^^ct. Narrative detaii should be kept to a 
minimum. Charts and graphs which present a clear piu.^.a of the main point of each indicator 
should do most of the wori<. Aim for a document which is short enough to be read easily in one 
sitting at the time it is received, so it wont be put aside for study later \vhen there is more time 
available.'* Ail too often, documents put aside for "later stuay are actually never seen again. 

The difficulty with making the report short and direct is that it entails risk of dangerous 
oversimplification. A "technical note" section for each indicator can be used when additional 
explanatory material is needed. 
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INDICATOR H 

Prevalence of tJcohol use in the last 30 days among school agful youngstcra» grades 8 ami 
1 1, in reporung schools as contajncU in Ppig Use bv Ore^um Pubiu: School SludcnlS. 



Description 
of indicator 



TECHNICAL NOT! 



Information about 
data source, 
^ reliability, etc. 



Betau&e alcohol is an illegal drug tor juveniles, its use is included as (*n indicator. The 
prevalence in the last 30 days is the indicator selected tor the same ieasc;j identitied lor 
Indicator «5. 
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Student Alcohol Use in Last 30 Days 
Grades 8 and 11 



Pareant of Studanta 



eth Qradara 




11th Qradara 



Portland 



1966! 
1988' 



46.5 
514 



Stat«wida 



36.3 
36.6 




Portland 



66.6 

62.1 



Statawida 



60.6 

6U3 



I 1966 



1 1066 



Soufc* *Diug u«t by Oftgon Fublic 
School** 



Indicator shown in 
graph format 



REMARKS: 



Short explanatory 
statement 



Eighth graders show an increase in use of alcohol in Ihe last 30 days in the Ponla,xl area 
and a slight decline lor the rest of the state. With older iiudenis 1 1 Ih graders - there is 
a slight decline tliroughout the state tn alcohol use but a more significant decrease among 
the Portland population. 
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Exhibit 9 



Dissemination 

Timing is an important element in achieving maximum public exposure for the release of the 
community impact indicator report. Arrange for a coordinated release to news media which 
considers, if possible, the daily deadline schedule for both print and electronic media. Choose a 
day when the press and public wont be preoccupied with other pre-scheduled events. (Some of 
the events which influence news coverage decisions can be anticipated: annual community 
festivals, important elections, etc.) 

Establish a time and place to present the rf ; )rt in a public forum, such as a public meeting of 
the community coalition membership. Give advance notice to the press. Provide an indication of 
the nature of the report, but don't "give It all away" prematurely.* The formal presentation can be 
followed by a news conference. Also, insure that key spokespersons are available for interviews as 
needed. 

It is also important to have discussion of the report on the agenda of every (public or private) 
organization with a potential role in the community prevention effort, fwlembers of the prevention 
coalition who prepared the report are the natural first choice emissaries to take the report back to 
the organizations which they represent. Getting the information received and understood in as 
many forums as possible increases the likelihood that policies and programs which contribute to 
prevention across a broad community front will be encouraged and supported. 



Action Outcomes 

Results of the community indicator project may be observed both within the coalition and in the 
community. 



•Full details-text of the announcement, the report itself, a descriptive press release-may be 
provided ahead of time on a "not for release before (date)" basis, assuming that a working 
relationship with the media has already been established. 
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Within the Coalition 

Work on the community indicator project can be expected to influence the direction of future 
projects undertaken by the coalition. The group, or Indivlduai members, may develop a particular 
interest in a topic explored as a possible impact indicator (babies bom to dmg abusing mothers, 
for instance), and follow through with addittonal efforts to take positive steps in that area. The 
energy thus mobilized is beneficial both to the community and to the coalition group internally, 
since the energy and eHectiveness of a group is a function of the energy and dedication of its 
individual members. 

Since the indicator project involves information gathering from many sources, it will tend to 
dr«w in additional people as the work proceeds. Mutual eHort between information seekers and 
infonnation providers to discover appropriate community indicators will widen the circle of persons 
involved and may provide new members for the group. 

Tne project's product, the report itself, may help serve as a recruitment tod for the community 
coalition group, by documenting the need for the group's work and demonstrating its ability to 
produce an important, useful, and tangible result while raising the awareness of potential members 
and the community as a whole. 



Community Results , 

The process of following up on the indicator project with actions aimed at specific policy and 
program effects in the fonnai and informal front line organizations, institutions, and social networks 
of the community is the larger task of the prevention coalition as a whole. As such, a complete 
discussion of this process is beyond the scope of this handbook. In brief, the work entails 
identifying key policy and program issues, targeting fomms in which these issues are decided, and 
developing a program of community prevention, and prevention advocacy, involving a wide range 
of community participation. 




Uses and limits in evaluation of community prevention 

Community prevention Is a long-temi effort. As the effort proceeds, the community irxJIcators 
will need to be updated at appropriate Intervals. Trends may thus beg\n to be observed In the 
nature and severity of the communitys alcohol and other drug problems, ideally, the trends will 
reflect the efforts of the prevention coalition, by showing a steady decline In adverse Impacts from 
substance abi/se. Many factors can Influence these overall Indicators, however, and iTwiny 
problems will likely persist In some form. The coalition will need to evaluate its specific projects by 
targeting outcomes which can be related directly to the activities or sen/ices provided. 
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Appendix A 

Further Reading About Community Coalitions in Alcohol and 

Abuse Prevention 
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Crowley, J.F. Mance For Change: . \ Plan For Community Action on Adolescent Drug 
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Disaggregating Profile Data: An Illustrative Example 
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Use of County Alcohol and Drug Treatment Data as Community 
Impact Indicator: Complications and Potential Benefits. 

The Regional Drug Initiative's committee on community impact assessment considered use of 
Mdtnomah County drug and alcohol treatment data as one of several possible indicators of tiie 
extent of drug and alcohol problems in the region. The group concluded that the available 
information about drug and alcohol treatment did not represent a suitable indicator for two ma; jr 
reasons: 

1 . 1 he number of people treated, and the relative size of the groups treated for alcohol 
problems or drug problems, largely reflected the capacity limits of the treatment programs. 
The number of persons inneesJ of treatment for alcohol or drug problems could not 
necessarily be inferred from the number treated. Further, since the funding formulas for 
drug and alcohol treatment are different in Oregon, with agencies receiving more money 
per '=slot" (defined as treatment for one person for one year) for drug treatment than for 
alcohol trec'.ment, agencies had an incentive to "convert" alcohol slots into drug slots. 

2. The available data only reflected publicly-funded treatment. The number of private 
treatment agencies and practitioners had increased rapidly, but private sector providers 
were not included in the reporting system. Thus the number of privately-treated clients 
was unknown, but presumed to be increasing 

The problems posed by these fa'^tors can be seen in Figure 1 , which shows the absolute and 
relative size of the groups treated for alcohol and drug problems in 1987. 1988, and 1989. The 
number treated for alcohol problems declined in 1988, then remained the same in 1989. 
Meanwhile, the number treated for drug problems increased in 1988 and again in 1989. It is not 
necessarily correct, however, to conclude that the number of individuals needing drug abuse 
treatment increased in proportion to the number treated, nor that the number needing treatment 
for alcohol problems did not change. The extent to which these changes are simply an artifact of 
policies and procedures of funding and administration cannot be determined from the data. Also, 
changes in the availability of private-sector treatment complicate the interpretation of the change in 
the overall number of clients treated. The total number seived increased, but the rate of increase 
in need is masked by the capacity limits of the programs and lack of data from private providers. 
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Thousands 




1987 1988 

Fiscal year ending June 30 



1989 



Alcohol 



Drug cm Total 



Combtntd acroaa aarvtca atamant. 
Undupttcatad within aarvica alamant but 
not batwaan barvlca alamanta. 



Figure 1. Drug and Alcohol Treatmant, Persona Served: Multnomah County 



Despite these shoficomings. a case can be made for including this indicator as an element of a 
community Impact Inc'ex. As we have seen, there are problems and complications with many 
potential sources of community data. These problems and complications are not necessarily fatal, 
however, provided that the strengths and weaknesses are understood, and potential users of the 
Information can somehow be deterred from making conclusions which the data cannot reasonably 
support 

Another Impact Indicator selection committee couW look at a similar set of facts and deckle to 
Include a treatment data Indteator. In spKe of the problems noted, on the grounds that such data 
provWes the only avatoble data on the actual size of the populatkjn being treated. The Informatton 
that more than 8.000 persons were ueated in county^nonitored programs In 1989 (with an 
unknown but presumably Increasing number treated privately) may not be an accurate Indicator of 
the number of people In need, but It helps provWe a sense of scale which might be helpful to 
members of the coalltton and puWte. The Increasing number of drug clients may or may not 
hdteate the size of the Increase of the drug^buslng populatton. but It does reveal something about 
the minimum size of the group and the probable direction of the trend. 
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The county treatment data can be used to ilustrate how indicator data might be analyzed in 
greater detal In order to better understand the Information and help the coalition in Its task of 
recommending steps which might lead to Improvements In the overall coordination of resources 
having to do with drug and nicohol problems. 

For example. Figure 2 provides a more detaled breakdown of the infonrottor already 
presented. Data for 1987. 1983. and 1989 Is plotted for each of 6 major 'seivtee element- 
categories. This analysis reveals that the changes m overall client populatkxi observed in Rgure 1 
tend to be concentrated in the W (detox) category, with the dnjg detox element growin j from 
zero to nearty 1 .000 persons served, whie alcohol detox declined from about 1 .700 to about 1 .1 00. 
The other servtee elements shown are resWentlal (Res) and outpatlerti (Otpt). 
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1000 
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Ale Res Drug Res Ale Dtx nrug Dtx Ale Otpt Drg Otpt 
Service Element 



1987 



1988 CD 1989 



Figure 2. Alcohol and Drug Treatment: By Service Element and Year, Peraone served: 
MuHnomah County 



Figure 3 shows that a large and Increasing share of the clients sen/ed by the county-monKored 
programs are persons with little or no income. Since the programs tradWonally generate revenue 
from a comblnatton of puWte funds and fees charged to clients according to an 'aMIty to pay- 
sliding scale. It appears that the potential for revenue from clients w» can afford to pay for 
treatment Is shrinking, whie the total demand for seivtee continues to grow. (The extent to which 
this reflects mlgratk)n of non-indigent clients to private programs Is not certain, but increased 
avaiiablity of private treatment Is a likely factor.) 
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■i Alcohol H Drug 

i 

Figurt 3. Incomt Undtr $250/month, MuHnomah County A ft 0 Client* 

Facts such as those Illustrated above can assist the committee in focusing on key issues of 
public policy and community knowledge In order to parttelpaie effectively In the ongoing process 
of fashtoning an effective and comprehensive response to substance abuse and related problems. 
Since there is virtually an unlimited variety of ways to break down and analyze any large data set. 
the group shouW seek out and rely upon expert advtee in Its chotee of questions to pursua. thus 
helping to ensure that the informatkxi value of the results is worth the time and effort expended in 
otHalning it. 
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EMMutfvtDtrtctor 



NORTHWEST REGIONAL EDUCATIONAL LABORATORY 

The ^telthw«st Regional Eduatlontl Laboratory (hWRHD is an independent, nonprofit research 
and development InsUtutlon established in 1966 to hrip others improve outcomes for chlldwm, 
youth, and adults by providing R&D assistance to schools and communities in providing equluble, 
high quality educational programs. I^REL provides assistance to education, government, 
community agencies business, and labor by: 

• Developii«anddliieminaUngelhctiveeducit^ 

• Cdndudifimeard) on eduational needs and pmMems 

• Piovidngtechnicaiaiilttanceineducilionalprobl^ 

• Evalurtngelhct t vene ss ofeducationiipfogratman^ 

• PiovMlngtnJnlf^ in educational planning, man ig em si » evalurton, and Imtmction 

• ScnHwasaninfemMtionresoufoeonefK^tteducationalpf^^ 

including netvvoridng among educational agendei^ initituUoni, and indviduali in the region 
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